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About  the  ATP  Economic  Assessment  Office 


The  Advanced  Technology  Program  (ATP)  is  a partnership  between  government  and  private 
industry  to  conduct  high-risk  research  to  develop  enabling  technologies  that  promise  significant 
commercial  payoffs  and  widespread  benefits  for  the  economy.  Since  ATP’s  inception  in  1990, 
the  Economic  Assessment  Office  (EAO)  evaluates  the  program  to  determine  the  returns  to  the 
taxpayer.  To  assess  whether  ATP  is  meeting  its  stated  objectives,  ATP’s  Economic  Assessment 
Office  employs  statistical  analyses,  case  study,  econometric,  survey,  and  other  methodological 
approaches  to  measure  program  effectiveness  by  evaluating  the  success  of  individual,  clusters, 
and  the  portfolio  of  ATP  projects  in  terms  of: 

• Inputs  (the  funding  and  staff  necessary  to  move  the  R&D  effort  forward) 

• Outputs  (project  research  results) 

• Outcomes  (new  products,  processes,  and  services  resulting  from  the  innovation) 

• Impacts  (longer-run  impacts  on  industries,  society,  and  the  economy) 

EAO  is  exploring  and  testing  the  use  of  new  methods  such  as  analysis  of  networks,  examining 
hot  spot  technologies  using  sophisticated  patent  citation  analysis,  determining  indicators  of 
regional  innovative  capacity,  understanding  the  role  of  universities  in  spinning  out  companies, 
and  exploring  different  approaches  for  reporting  benefit-cost  results. 

Key  features  of  ATP’s  evaluation  program  include: 

• The  Business  Reporting  System,  a unique  online  survey  of  participants,  which  gathers  data 
on  an  annual  basis  on  the  business  progress  and  indicators  of  future  economic  impact  of 
funded  projects  and  on  a biennial  basis  up  to  six  years  post-project  completion. 

• Special  surveys,  including  the  Survey  of  Applicants  that  addresses  the  counterfactual 
question  of  what  happens  when  projects  are  not  funded,  and  the  Survey  of  Joint  Ventures, 
a one  time  survey  of  joint  venture  projects  funded  between  1990  and  2001,  designed  to 
understand  the  benefits  of  alliances. 

• Status  Reports,  mini-case  studies  of  completed  projects  several  years  out.  Projects  are 
rated  on  a zero  to  four  star  scale,  representing  a range  of  performance  from  poor  to 
outstanding  on  technical  success,  knowledge  generation  and  diffusion,  commercial  entry, 
and  future  outlook. 

• Benefit-cost  studies,  which  identify,  assess,  and  quantify  the  net  private,  public,  and  social 
benefits  of  ATP  project  outcomes. 

• Economic  and  policy  research  studies  that  examine  particular  aspects  and  impacts  of  the 
program,  including  the  effect  of  collaboration  on  the  research  productivity  of  participating 
organizations  and  the  role  of  the  program  in  the  U.S.  innovation  system. 

Contact  ATP’s  Economic  Assessment  Office  for  more  information: 

• On  the  Internet:  http://www.atp.nist.gov/eao/eao_main.htm 

• By  e-mail:  atp-eao@nist.gov 

• By  phone:  1-301-975-8978,  Stephanie  Shipp,  Director,  ATP’s  Economic  Assessment  Office 
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Abstract 


♦ 


The  Economic  Assessment  Office  of  the  Advanced  Technology  Program  (ATP)  measures  the 
economic  impact  of  co-funding  projects  with  the  private  sector  to  develop  high-risk,  enabling 
technologies,  and  seeks  to  increase  understanding  of  the  underlying  relationships  between 
technological  innovation  and  economic  phenomena.  The  National  Academy  of  Sciences  has 
praised  ATP’s  evaluation  program  as  “one  of  the  most  rigorous  and  intensive  efforts  of  any 
U.S.  technology  program”  (National  Research  Council,  2001).  ATP’s  evaluation  efforts  were 
instituted  to  meet  external  requests  for  ATP  program  results,  to  use  evaluation  as  a management 
tool  to  meet  program  goals  and  improve  program  effectiveness,  to  understand  ATP’s  contribution 
to  the  U.S.  innovation  system,  and  to  develop  innovative  methodologies  to  measure  the  impact 
of  public  RND  investment.  To  do  this,  ATP  economists  track  progress  throughout  each  project’s 
life  and  into  the  post-project  period  by  conducting  surveys,  compiling  data,  producing  statistical 
analyses,  undertaking  economic  studies,  and  commissioning  studies  with  consultants  and  research 
economists.  This  paper  describes  the  evolution  of  ATP’s  evaluation  activities,  its  evaluation  best 
practices  based  on  ATP’s  experience  since  1990,  and  findings  from  ATP  studies. 
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ATP  Mission 


The  mission  of  the  Advanced  Technology  Program  (ATP)  is  to  accelerate  the  development  of 
innovative  technologies  for  broad  national  benefit  through  partnerships  with  the  private  sector. 
ATP  began  in  1990  in  response  to  the  erosion  of  U.S  industry’s  international  competitiveness 
in  strategic  markets  and  the  relative  slowness  of  U.S.  firms  in  translating  inventions  created 
in  universities,  national  laboratories,  and  corporate  laboratories  into  innovative  products  and 
processes  (National  Research  Council,  1999;  Ruegg  and  Feller,  2003,  U.S.  Congress  1988).  ATP 
provides  cost-shared  funding  to  industry  to  accelerate  the  development  and  broad  dissemination 
of  challenging,  high-risk  technologies  that  promise  significant  commercial  payoffs  and  wide- 
spread benefits  for  the  nation.  This  unique  government-industry  partnership  helps  companies 
accelerate  the  development  of  emerging  or  enabling  technologies.  Those  technologies,  in  turn, 
lead  to  revolutionary  new  products  and  to  new  industrial  processes  and  services  that  can  com- 
pete in  rapidly  changing  world  markets. 

ATP  challenges  industry  to  take  on  projects  that  have  higher  technical  risk,  and  commensuratelv 
higher  potential  payoffs  for  the  nation,  than  they  otherwise  would  pursue.  The  ATP  project 
selection  criteria  reflect  this  philosophy.  Half  of  the  criteria  are  based  on  scientific  and  tech- 
nological merit  and  include  an  explanation  of  the  innovation,  a detailed  research  plan,  and 
justification  that  the  approach  is  feasible  and  has  the  potential  to  overcome  the  technical 
hurdles.  The  other  half  of  the  criteria  are  based  on  the  potential  for  broad-based  economic 
benefits,  including  benefits  to  the  economy  and  society  that  would  result  from  developing  the 
new  technology,  justification  for  the  need  for  ATP  funding,  and  a plan  for  how  the  technology, 
once  developed,  will  be  commercialized.  Proposals  that  are  submitted  in  response  to  ATP's 
announced  competitions  are  peer-reviewed  against  the  published  selection  criteria.  On  average, 
one  out  of  eight  proposals  meet  ATP’s  criteria  for  funding. 

ATP  accepts  applications  from  single  companies  and  joint  ventures.  For-profit  companies  may 
apply  as  single  applicants  to  receive  an  award  for  up  to  $2  million  over  three  years  to  cover 
project  costs.  Single-company  applicants  are  required  to  cover  their  indirect  costs — this  require- 
ment encourages  the  participation  of  small  firms  that  have  low  overhead  costs.  In  fact,  small 
businesses  are  thriving  in  the  program,  (nearly  half  of  all  ATP-funded  small  firms  have  fewer 
than  20  employees)  and  lead  two  out  of  three  of  all  projects.  Single-company  applicants  often 
bring  in  subcontractors  (universities  or  other  companies)  to  participate  in  the  project.  Large, 
Fortune  500  companies  applying  as  single-company  applicants  must  cover  at  least  60  percent 
of  total  project  costs — this  requirement  encourages  large  firms  to  formally  collaborate  with 
others  and  apply  as  a joint  venture. 

At  least  two  separately  owned  for-profit  companies  may  apply  as  a joint  venture,  with  both 
companies  substantially  contributing  to  the  research  effort  and  to  the  requirement  to  cover  at 
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least  half  of  the  total  project  costs.  Additional  organizations  (universities,  nonprofits,  or  other 
companies)  may  join  the  joint  venture  either  as  formal  participants  or  as  subcontractors.  Joint 
ventures  can  receive  ATP  funding  for  up  to  five  years,  with  no  funding  limitation  other  than  the 
announced  availability  of  funds  and  the  organizations'  ability  to  cover  half  the  total  project  costs. 

ATP  announces  competitions  through  the  Federal  Register  and  held  44  competitions  between 
1990  and  September  2004.  ATP  has  provided  $2.2  billion  in  awards,  and  industry  has  provided 
an  additional  $2.1  billion  as  cost  share,  for  a total  of  $4.3  billion  for  high-risk  research.  Of 
the  768  projects  awarded  to  date,  550  are  to  single-company  applicants  and  218  are  to  joint 
ventures.  More  than  165  universities  and  30  national  laboratories  participate  in  ATP  projects, 
reflecting  the  collaborative  nature  and  the  diversity  of  the  projects’  participants.  Projects  that 
are  awarded  are  organized  into  four  broad  technology  areas:  advanced  materials  and  chemistry, 
information  technology,  electronics  and  photonics,  and  biotechnology.  Manufacturing  is  a subset 
in  all  four  categories.  Technical  topics  under  each  of  the  broad  categories  are  broad  and  diverse. 
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Measuring  Against  Mission: 
An  Evaluation  Framework 


Evaluation  has  been  an  integral  part  of  program  operations  from  the  outset.  To  learn  about  the 
program’s  impact,  ATP  set  aside  a small  amount  of  their  initial  budget  in  1990  to  fund  rudimen- 
tary evaluation  activities.  Since  then,  the  budget  for  program  evaluation  has  grown  significantly, 
as  has  interest  in  evaluation.  With  a professional  staff  of  economists,  statisticians,  information 
specialists,  social  scientists,  business  liaison  specialists,  and  administrative  support,  the  ATP’s 
Economic  Assessment  Office  (EAO)  is  charged  with  carrying  out  ATP’s  evaluation  activities. 

Ei\0  aims  to  measure  the  economic  impacts  of  ATP’s  funding  of  high-risk,  enabling  technologies, 
and  also  to  increase  understanding  of  underlying  relationships  between  technological  change  and 
economic  phenomena.  EAO  also  provides  business  and  economic  expertise  for  ATP  selection 
boards  and  locates  expert  business  reviewers  to  review  proposals. 

ATP’s  evaluation  program  has  four  main  goals: 

1.  To  meet  external  requests  for  ATP  program  results, 

2.  To  use  evaluation  as  a management  tool  to  meet  program  goals  and  to  improve  program 
effectiveness, 

3.  To  understand  ATP’s  contribution  to  the  U.S.  innovation  system,  and 

4.  To  develop  innovative  methodologies  to  measure  the  impact  of  public  R&D  investment. 

ATP’s  Economic  Assessment  Office  tracks  progress  throughout  the  life  of  funded  projects  and 
for  several  years  after  the  ATP  funding  ends.  Evaluation  work  consists  of  conducting  surveys, 
compiling  data,  producing  statistical  analyses,  undertaking  economic  and  policy  research  studies, 
and  commissioning  studies  by  consultants  and  research  economists. 

Evaluation  works  best  when  it  is  closely  mapped  to  a program’s  mission.  ATP’s  legislative  man- 
date is  to  increase  the  prosperity  of  the  Linked  States  by  funding  the  development  of  high  risk 
technologies  through  a public-private  partnership.  ATP’s  goals  are  to  add  to  the  nation's  scientific 
and  technical  knowledge  base,  to  foster  accelerated  technology  development  and  commercializa- 
tion, to  promote  collaborative  R&D,  to  refine  manufacturing  processes,  to  ensure  small  business 
participation,  to  increase  the  competitiveness  of  U.S.  firms,  and  to  generate  broad  economic  and 
social  benefits  (Ruegg  and  Feller,  2003). 

ATP  award  recipients  deliver  benefits  directly  and  indirectly.  Direct  benefits  are  achieved  when 
technology  development  and  commercialization  is  accelerated,  which  leads  to  private  returns 
and  market  spillovers.  Indirect  benefits  are  delivered  through  publications,  conference  presenta- 
tions, patents,  and  other  ways  in  which  knowledge  is  disseminated.  From  program  purpose  and 
design  to  final  outputs,  outcomes,  and  impacts,  ATP’s  evaluation  program  measures  these  direct 
and  indirect  benefits. 
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ATP's  Evaluation  Best  Practices 


ATP’s  experience  in  funding  early-stage  technologies  and  evaluating  the  impact  of  its  awarded 
projects  has  resulted  in  many  best  practices.  These  best  practices  may  prove  useful  to  similar 
government  programs  in  the  early  stages  of  their  operations  or  to  government  programs  that 
must  meet  external  performance  reporting  requirements. 


COMMITTING  TO  PERFORMANCE  EVALUATION 

One  of  the  most  important  best  practices  is  to  establish  the  practice  of  evaluation  and  to  sustain 
those  activities  despite  budgetary  pressures.  ATP  allocates  funding  for  a staff  dedicated  to  eval- 
uation activities — the  Economic  Assessment  Office — and  for  carrying  out  evaluation  activities 
using  internal  and  external  resources.  It  is  important  for  public  research  and  development  pro- 
grams to  treat  evaluation  as  a core  activity  and  to  pursue  evaluation  within  a framework  that 
measures  the  program  against  its  stated  objectives.  Having  a dedicated  staff  with  appropriate 
backgrounds,  capabilities,  and  experience  is  essential.  Having  a dedicated  budget  for  evaluation 
activities  is  critical. 


USING  A MULTIFACETED  APPROACH  TO  EVALUATION 

Evaluation  of  early  stage  technologies  and  their  longer  run  impacts  requires  a multi-faceted 
approach  to  evaluation  because  developing  new  technologies  is  complex  and  can  take  several 
years.  Our  evaluation  tools  assess  commercialization  as  well  as  knowledge  creation  and  dissem- 
ination. These  methods  must  accommodate  the  measuring  of  inputs,  outputs,  outcomes,  and 
impacts  over  the  life  cycle  of  a project.  Research  and  development  take  place  in  the  short  to  mid 
term,  commercialization  in  the  mid  to  longer  term,  and  widespread  diffusion  of  the  technology 
over  a longer  time  horizon.  This  time  frame  varies  by  technology  area — shorter  for  information 
technology  projects  and  much  longer  for  biotech  projects  (Powell  and  Moris,  2002).  It  also 
accounts  for  why  multiple  evaluation  approaches  are  needed  to  capture  the  status  of  projects 
at  various  stages  of  their  life  cycle. 


COMMISSIONING  EXTERNAL  STUDIES  BY  EXPERTS 

ATP  contracts  with  experts  to  conduct  economic  analysis  of  individual  projects,  clusters  of  proj- 
ects, or  concepts  underlying  the  economic  principles  of  the  program.  ATP’s  Economic  Assessment 
Office  works  with  well-known  researchers  to  shape,  manage,  and  produce  many  of  our  reports. 
In  the  early  years,  ATP’s  Economic  Assessment  Office  worked  with  economists  affiliated  with  the 
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National  Bureau  of  Economic  Research  to  help  lay  a strong  foundation  for  evaluating  the  pro- 
gram. Zvi  Griliches,  Edwin  Mansfield,  Adam  Jaffe,  Bronwyn  Hall,  and  others  collaborated  with 
ATP  on  important  research  to  explore  how  to  measure  and  track  key  economic  concepts  that 
apply  to  government  support  for  the  development  of  high-risk,  enabling  technologies  carried 
out  by  the  private  sector.  They  studied  concepts  such  as  spillovers  (knowledge,  network,  and 
market  spillovers — see  Jaffe,  1997),  return  on  investment  (social,  private,  and  public  rates  of 
return),  (see  Mansfield,  1996)  and  research  productivity  (see  Darby  et  ah,  2002  and  Sakakibara 
and  Branstetter,  2003). 


EVALUATING  UNSUCCESSFUL  PROJECTS 

Another  best  practice  is  evaluating  unsuccessful  projects  along  with  successful  ones.  There  is 
a great  deal  to  learn  from  projects  that  failed  to  complete  their  goals  or  to  deliver  promised 
benefits.  ATP  has  analyzed  the  reasons  behind  projects  terminating  early  (see  ATP,  2001, 
Appendix  B).  The  knowledge  generated  by  examining  the  reasons  why  projects  fail  can  enhance 
project  selection  and  project  management. 

Almost  10  percent  of  projects  terminate  early.  A project  can  end  early  or  not  start  for  partici- 
pant-initiated reasons,  such  as  a change  in  goals,  financial  distress,  lack  of  technical  progress, 
or  the  inability  of  a joint  venture  project  to  reach  an  agreement  on  rights  to  intellectual  property. 
A project  can  also  end  for  ATP-imtiated  reasons,  such  as  the  project  no  longer  meets  ATP 
project  selection  criteria  or  it  shifts  away  from  the  pursuit  of  high-risk  research.  In  a very  few 
cases,  early  success  was  the  cause  for  early  termination. 


STRATEGICALLY  PRESENTING  RESULTS 

Results  have  more  effect  if  they  are  presented  so  that  a nontechnical  person  can  understand 
the  science  and  commercialization.  Results  are  presented  in  multiple  ways — a brief  abstract 
enabling  someone  to  quickly  grasp  the  key  findings,  an  executive  summary  for  someone  who 
wants  an  overview  of  key  highlights,  and  the  full  report.  Quantitative  findings  are  presented  in 
tabular  form,  with  graphics,  and  with  accompanying  qualitative  analyses.  We  release  many  of 
our  findings  in  fact  sheets  and  make  them  available  on  ATP’s  website.  We  have  also  published 
three  special  topic  brochures  that  highlight  projects  in  the  health  care,  energy,  and  manufacturing 
sectors.  (See  ATP,  2003b,  2003c,  2005.) 

Another  way  that  results  and  data  are  summarized  is  in  the  form  of  a ‘statistical  abstract,’  an 
idea  borrowed  from  the  U.S.  Census  Bureau’s  annual  Statistical  Abstract.  Plans  are  to  publish 
the  ATP  statistical  abstract  every  other  year — the  first  was  released  in  September  2004,  in  a 
report  called  Measuring  ATP  Impact , 2004  Report  on  Economic  Progress.  The  report  describes 
ATP,  using  findings  and  data  from  recent  reports  and  statistics.  It  also  provides  summaries  of 
recent  studies  and  ten  detailed  statistical  tables  that  provide  data  on  number  and  distributions 
by  types  of  awards,  technology  areas,  geographic  regions,  university  participation,  number  of 
patents,  commercialization,  and  post-award  attraction  of  external  funding.  The  data  are  pre- 
sented for  all  projects  and  by  characteristic  of  the  project. 
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DEVELOPING  INNOVATIVE  METHODS  TO  EVALUATE  ATP'S  EFFECTIVENESS 

Evaluation  of  emerging  technologies  is  a relatively  new  field.  While  traditional  economic  and 
social  science  methods  can  be  employed  to  assess  program  success,  the  existing  tools  are  often 
insufficient  to  describe  the  nuances  and  input  of  public-private  investments.  It  is  appropriate  to 
modify  existing  tools,  develop  exciting  new  tools,  or  combine  existing  methods  in  ways  never 
before  explored. 

For  example,  one  of  the  more  difficult  concepts  to  measure  is  social  return  resulting  from  an 
ATP  project.  Social  return  includes  private  returns  to  the  participating  company  in  the  project, 
and  public  returns,  including  knowledge,  network,  and  market  spillover  benefits  to  that  com- 
pany’s customers  or  to  other  firms,  and  a variety  of  indirect  benefits  to  other  companies  and 
their  customers  as  a result  of  the  diffusion  of  knowledge  created  from  the  project.  (See  Jaffe, 
1997;  Chang,  Shipp,  and  Wang,  2002,  for  an  historical  description  of  this  issue.) 

Despite  the  difficulties  in  measuring  social  return,  ATP  has  pursued  a greater  understanding 
of  this  concept  by  collaborating  with  consultants,  professional  economists,  and  academicians. 
Together,  we  carry  out  prospective  benefit-cost  studies  of  a range  of  technologies  and  projects 
to  test  and  stretch  various  methodological  approaches.  These  studies  include  case  studies  of 
projects  that  developed  closed-cycle  air  refrigeration  technology  (Pelsoci,  2001),  flow-control 
machining  technology  (Ehlen,  1999),  and  technologies  that  reduced  the  dimensional  variation 
of  LhS.  motor  vehicles  (Polenske  et  ah,  2004).  By  supplementing  core  in-house  evaluation  capa- 
bility with  expertise  provided  by  outside  contractors,  ATP  has  pursued  a balanced  approach  to 
evaluation  and  has  welcomed  new  ideas  and  approaches — another  example  of  a best  practice. 

In  measuring  spillovers,  for  example,  we  have  used  various  approaches  and  means  of  illustration. 
To  capture  knowledge  spillovers,  for  the  status  reports  of  completed  projects — a portfolio-wide, 
mini-case  study  tool — we  developed  patent  trees,  which  illustrate  multi-tiered  citations  of  patents 
that  were  issued  for  ATP-funded  technologies.  In  addition,  we  commissioned  a study  to  examine 
knowledge  spillovers  using  social  network  analysis.  This  emerging  method  uses  fuzzy  logic  and 
systems  analysis  to  examine  knowledge  spillovers  from  research  and  development  projects  within 
networks  of  participating  organizations.  (See  the  discussion  in  Ruegg  and  Feller,  2003,  pp. 
271-75;  Fogarty  et  ah,  2005.) 

To  study  market  spillovers,  we  have  recently  begun  exploring  the  use  of  the  U.S.  Department 
of  Commerce’s  Bureau  of  Economic  Analysis  input-output  tables.  Specifically,  we  have  taken 
the  first  50  completed  ATP  projects  and  mapped  their  make-and-use  industries  to  trace  where 
the  new  technologies  began  and  where  they  have  since  ended  up  (Popkin,  2003).  We  are  also 
exploring  other  emerging  methods  to  measure  spillovers  and  the  impact  of  ATP  funding, 
including  coding  potential  commercial  applications  identified  by  ATP  project  participants  using 
NAICs  (North  American  Industry  Classification)  codes  to  identify  make-and-use  industries 
that  illuminate  the  spillover  path  (Nail  and  Brown,  2005,  forthcoming). 


USING  SYSTEMATIC  DATA  COLLECTION 


Perhaps  the  cornerstone  of  AT  P’s  evaluation  program  is  its  comprehensive  survey  and  data 
collection  system.  Our  survey  collection  efforts  are  structured  to  align  with  our  overall  evalua- 
tion goals,  which  in  turn  are  crafted  to  optimize  the  performance  of  ATP.  As  part  of  an  ongoing 
survey  and  database  assessment  effort,  we  have  identified  five  broad-based  goals  that  form  the 
conceptual  basis  of  our  surveys:  (1)  opportunities  for  national  economic  benefits,  (2)  acceleration 
of  R&D,  (3)  increased  investment  in  high-risk,  long-term  technology,  (4)  stimulation  of  collabo- 
ration, and  (5)  progress  in  commercialization  of  technology.  These  five  goals  define  how  ATP 
projects  affect  the  economy  and  society. 


ATP  Surveys 


As  mentioned  above,  systematic  data  collection  is  one  component  of  ATP’s  best  practices.  This 
section  provides  an  in-depth  explanation  of  how  ATP  surveys  have  evolved  and  continue  to 
improve. 


CONCEPTUAL  BASIS  OF  ATP  SURVEYS 

Initiated  in  ATP's  early  years,  the  survey  program  has  continually  incorporated  refinements  in 
structure,  content,  and  form  to  ensure  that  they  remain  effective  and  informative  tools  for  the 
assessment  of  ATP  progress  toward  its  goals. 

From  their  inception  in  1991  as  telephone  surveys,  ATP's  surveys  have  evolved  through  several 
distinct  stages  of  development.  This  evolution  has  occurred  as  we  (1)  have  gained  a deeper 
understanding  of  the  economic,  social,  business,  and  technological  processes  in  which  ATP 
participates;  (2)  incorporated  feedback  from  prior  evaluation  efforts,  including  surveys,  economic 
and  policy  research  studies,  and  benefit-cost  analyses;  and  (3)  integrated  state-of-the-art  survey 
methodologies  and  techniques  into  our  analytical  and  administrative  structures. 

ATP  surveys  can  be  viewed  as  a microcosm  of  our  overall  evaluation  program  at  ATP.  The 
survey  system  is  a multifaceted  effort  that  is  designed  to  meet  multiple  (but  complementary) 
program  goals  while  balancing  efficiency  in  operation  with  excellence  in  results.  This  is  achieved 
by  identifying  and  leveraging  both  internal  and  external  resources  and  harnessing  the  benefits 
of  collaboration  with  survey  experts  (a  tactic  learned  through  our  evaluation  of  ATP),  and  by 
relying  on  continual  self-assessment  and  feedback.  We  do  not  lose  sight  of  our  goal  to  measure 
against  mission,  in  the  short,  medium,  and  long  term. 

To  design  our  survey  content,  economic  and  organizational  changes  we  would  expect  to  observe 
in  our  award  recipient  population  if  progress  were  made  toward  those  goals  are  identified. 

We  then  crafted  survey  questions  to  capture  the  observed  behaviors  and  outcomes.  The  themes 
and  topics  defined  by  the  goals  are  reflected  in  multiple  lines  of  questions  that  vary  in  a logical 
progression  over  the  survey  lifetime.  Baseline  information  is  collected  on  the  initial  survey, 
and  follow-up  questions  in  each  area  are  included  at  the  appropriate  anniversary,  closeout,  or 
post-project  surveys.  Several  variants  of  the  surveys  for  different  types  of  organizations  in  the 
population  are  used.  For  example,  participating  nonprofit  organizations  or  universities  are  given 
a slightly  different  survey  than  companies  to  reflect  their  unique  roles  in  the  project  and  different 
organizational  structures. 
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Diffusion  of  knowledge  is  captured  by  tracking  patents,  papers,  presentations,  and  other  infor- 
mation about  intellectual  property.  Measures  of  social  and  environmental  effects,  or  spillovers, 
of  the  technologies  are  captured. 

Acceleration  of  R&D  is  measured  in  terms  of  reduced  time  to  achieve  technical  progress. 
Instances  in  which  the  technology  would  not  have  been  developed  at  all,  or  perhaps  on  a lesser 
scale  or  scope,  without  ATP  funding  are  identified.  This  captures  the  key  counterfactual  ques- 
tion: what  is  the  difference  made  by  ATP?  Reductions  in  time-to-market  are  also  tracked. 

The  goal  of  increased  investment  in  high-risk,  long-term  technologies  is  an  indicator  of  the 
extent  to  which  the  work  of  ATP  results  in  sustained  technological  development.  The  level  of 
R&cD  risk  is  collected  by  seeking  estimators  of  probability  of  success,  ambitiousness  of  goals, 
technical  difficulty,  degree  of  innovation,  and  project  duration.  The  “halo  effect,'”  the  extent  to 
which  the  initial  ATP  award  has  stimulated  additional  funding,  either  from  within  the  firm  or 
from  external  sources  is  also  captured.  (Survey  of  ATP  Applicants  2000,  (2003a);  Feldman  and 
Kelley,  2001;  Solomon  Associates,  1993.) 

The  right  mix  of  collaboration  can  lead  to  positive  outcomes  for  technology  creation,  develop- 
ment, and  commercialization,  and  for  the  national  benefits  that  arise  from  knowledge  diffusion 
and  new  commercial  products  and  processes.  To  determine  whether  and  how  collaboration  is  a 
factor  in  creating  such  impacts  among  our  award  recipients,  we  survey  to  learn  about  the  struc- 
ture and  nature  of  collaborative  arrangements.  And  because  our  primary  goal  in  the  Economic 
Assessment  Office  is  to  measure  the  impact  of  ATP,  we  are  interested  in  whether  the  collabora- 
tion would  have  occurred  at  all  without  ATP  funding. 

The  centerpiece  of  our  surveys  is  the  Business  Reporting  System,  or  BRS,  which  superseded  the 
original  telephone  surveys  in  the  mid-1990s  (Anderson  et  al.,  2003).  As  did  its  predecessor,  the 
first  BRS  addressed  the  dual  objectives  of  ATP  project  management  and  program  evaluation.  It 
also  incorporated  lessons  learned,  most  notably,  use  of  the  counterfactual  approach  to  measure 
the  overall  impact  of  ATP.  The  BRS  is  not  a single  survey  instrument,  but  rather  a system  of 
surveys  administered  to  all  ATP  participants  throughout  the  project’s  duration  and  beyond. 
Then,  as  now,  the  BRS  consisted  of  a series  of  surveys  conducted  at  baseline;  annually  upon 
each  project  anniversary  (up  to  five  years,  depending  on  the  type  of  project  award);  and  two, 
four,  and  six  years  after  project  closeout.  The  first  BRS  was  sent  to  participants  on  a diskette. 
Following  developments  in  survey  design  and  methodology,  we  transitioned  to  a web-based 
application  and  revised  the  survey  design  in  1999. 

National  economic  benefits  are  measured  by  gathering  information  about  business  growth, 
the  development  of  business  relationships  and  networks,  and  the  diversity  of  commercial 
applications  arising  from  the  technology  that  ATP  has  funded. 

Finally,  ATP  surveys  capture  commercialization  progress,  results,  and  expectations,  or  specific 
aspects  of  it.  Although,  ATP  does  not  fund  the  commercialization  phase  of  projects.  Our  mission 
at  ATP  is  not  simply  to  fund  high-risk  technologies,  but  to  fund  high-risk  technologies  that 
have  a strong  potential  to  enhance  economic  growth.  Economic  growth  can  only  be  achieved 
when  the  technology  enters  the  marketplace.  To  measure  this  impact,  information  is  collected 


10 


ATP  Survey 


from  firms  on  current  and  expected  economic  value  achieved  through  revenues  from  commercial 
applications  of  the  technology,  licensing,  and  cost  savings. 


SURVEY  AND  DATABASE  REFINEMENTS  AND  INTEGRATION 

Today,  ATP  is  in  the  midst  of  a conversion  to  another  iteration  of  our  surveys.  The  current 
transition  involves  three  components:  (1)  refinements  to  the  content  and  structure  of  the  survey 
instruments;  (2)  collaboration  with  an  external  professional  survey  services  firm  for  program- 
ming, administration,  and  support  for  survey  design  and  data  collection;  and  (3)  integration 
into  the  BRS  of  special  surveys  that  we  have  designed  and  conducted  in  recent  years. 

Refinements  to  the  survey  content  and  structure  are  being  made  by  incorporating  feedback  from 
prior  survey  and  evaluation  efforts,  then  by  assessing  proposed  survey  revisions  against  the  five 
conceptual  goals  outlined  above.  An  internal  survey  design  team,  working  in  conjunction  with 
an  external  survey  design  firm,  made  initial  revisions.  The  in-house  team  provides  expertise  in 
subject  matter  content,  past  practice,  program  goals,  and  overall  direction,  and  the  external 
contractor  provides  professional  guidance  in  survey  methodology.  The  contractor  will  ultimately 
program  and  conduct  the  revamped  system  of  surveys,  lending  efficiency  and  consistency  in 
the  long-term  administration  of  the  surveys,  enabling  us  to  focus  on  analysis  and  results.  The 
survey  organization  will  follow  up  with  respondents  who  do  not  answer  the  survey  questions 
or  answer  only  some  of  the  questions.  They  will  also  edit  the  data  for  consistency  and  to  reduce 
item  non-response. 

Assessment  of  the  revised  BRS  and  other  surveys  against  evaluation  program  goals  is  a long- 
term process  being  conducted  by  the  full  economic  assessment  staff,  led  by  the  survey  design 
team.  As  the  survey  design  team  completes  a draft  of  each  phase  of  the  revised  survey  instru- 
ment, the  team  submits  the  draft  to  the  staff  for  review  against  program  goals.  Staff  members 
provide  input  and  comments,  focusing  on  their  assigned  areas  of  expertise,  enabling  the  survey 
team  to  further  refine  the  draft. 

Because  the  BRS  is  a unified  system,  changes  to  later  stages  of  the  survey — such  as  the  project 
closeout  or  post-project  surveys — affect  the  baseline  or  anniversary  surveys,  necessitating  further 
modifications  to  earlier  drafts.  Although  we  expect  the  bulk  of  the  survey  redesign  work  and 
transition  to  the  contractor  for  administration  to  be  completed  by  late-2005,  we  additionally 
expect  the  survey  evaluation  and  revision  mechanism  to  be  an  ongoing  process. 

Though  not  a large  survey  in  terms  of  population  size,  the  BRS  has  a complex  structure. 
Including  32  ATP  awards  made  in  September  2004,  ATP  has  now  funded  768  projects  involving 
more  than  1,500  participants.  The  population  of  the  survey  is  diverse.  About  one-third  of  the 
projects  are  joint  ventures,  and  over  half  include  universities.  All  projects  involve  one  or  more 
for-profit  firms,  and  these  firms  vary  in  size  from  tiny  start-ups  to  large  Fortune  500  companies. 

Furthermore,  although  the  survey  is  administered  to  firms  (or  other  business  or  research  organi- 
zations), it  is  the  project  rather  than  the  organization  that  is  the  focal  point.  We  are  interested  in 
following  the  technology,  rather  than  the  firm  per  se,  although  we  are  interested  in  the  economic 
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and  institutional  effects  and  interactions  between  the  technology  and  the  firm.  Thus,  this  is  not 
strictly  an  establishment  survey. 

A layer  of  complexity  stems  from  the  survey  time  frame.  ATP  funding  is  awarded  through 
competitions,  but  the  competitions  do  not  occur  at  the  same  time  each  year  and,  especially  in 
recent  years,  competitions  may  consist  of  several  batches.  This  raises  the  question  as  to  how  to 
define  a cohort  and  complicates  survey  administration  and  analysis.  For  example,  the  27  projects 
that  were  awarded  funding  in  May  2004  were  actually  the  last  batch  of  award  recipients  from 
our  2002  competition.  They  were  the  first  group  to  begin  with  the  newly  redesigned  baseline 
survey  in  October  2004.  Furthermore,  project  duration  may  vary  from  about  two  to  five  years, 
and  individual  projects  may  be  terminated  early  or  suspended  temporarily,  requiring  an  adjust- 
ment to  survey  cycle  times. 

A component  of  the  current  survey  conversion  effort  is  the  integration  of  special  surveys  into 
the  BRS.  In  particular,  ATP  conducted  two  separate  surveys  in  recent  years:  a Survey  of  ATP 
Joint  Ventures  and  a Survey  of  ATP  Applicants.  The  Survey  of  ATP  Joint  Ventures  addresses 
the  structure,  nature,  and  impact  of  collaborative  relationships.  The  Survey  of  ATP  Applicants 
covers  the  full  population  of  all  firms  that  applied  for  ATP  funding  in  a given  competition, 
enabling  analysis  of  the  question  of  additionality  (or  the  difference  that  the  ATP  award  made) 
by  studying  both  firms  that  do  receive  ATP  funding  and  firms  that  do  not.  Incorporating  con- 
cepts and  questions  from  both  surveys  into  the  BRS  will  improve  efficiency  in  administration, 
facilitate  analysis,  and  reduce  respondent  burden.  Nonawardees  will  continue  to  receive  a sepa- 
rate survey,  since  the  BRS  is  designed  to  collect  data  from  awardees. 

Finally,  we  are  embarking  on  a redesign  and  integration  of  the  databases  that  underlie  our 
multifaceted  surveys.  We  also  hope  to  integrate  data  from  external  sources,  such  as  data  from 
Compustat  and  Dun  &c  Bradstreet,  into  our  internal  databases.  The  database  integration  effort 
complements  our  survey  redesign  work,  and  it  is  also  necessitated  by  the  growth  of  ATP. 
Database  methods  and  structures  created  in  the  early  years  are  no  longer  adequate  for  an 
expanding  survey  population,  especially  in  light  of  the  complex  structure  of  the  survey  cycles. 
Because  our  ultimate  goal  is  the  collection  and  production  of  data  to  serve  the  multiple  functions 
of  program  evaluation,  the  analysis  of  R&D,  and  the  identification  and  development  of  new 
research  and  evaluation  methodologies,  survey  and  database  development  will  remain  on  the 
agenda  for  continuous  process  improvement. 


Examples  of  ATP's  Studies 


ATP  uses  a multi-faceted  approach  to  evaluation.  Reflecting  the  richness  and  diversity  of  studies, 
several  examples  of  studies,  plus  highlights  of  findings  from  those  studies,  are  presented  below. 
These  examples  include  policy  studies.  Status  Reports  of  completed  ATP-funded  projects,  selected 
survey  results,  and  benefit-cost  studies.  These  examples  are  provided  to  show  the  depth  and 
breadth  of  our  assessment  work. 


DEFINING  ATP'S  ROLE  IN  THE  POLICY  ARENA 

In  a report  commissioned  by  ATP's  Economic  Assessment  Office,  Branscomb  and  Auerswald 
(2002)  address  the  question:  how  important  is  the  public  role  in  funding  early-stage  technology 
development?  They  estimate  that  between  $5  billion  (2  percent)  and  $36  billion  (14  percent)  of 
overall  national  R&D  spending  in  1998  was  devoted  to  early-stage  technology  development. 

The  relatively  small  share  of  total  national  R&T)  spending  devoted  to  early-stage  technology 
development,  which  arguably  drives  future  economic  growth,  supports  the  view  that  there  is  a 
“funding  gap”  in  which  the  amount  of  funding  currently  available  is  less  than  what  is  socially 
optimal  or  desirable. 

Of  particular  interest  is  the  finding  that  the  federal  government,  and  not  organized  venture 
capital,  is  a major  funding  source  for  early-stage  technology  development.  The  federal  govern- 
ment provided  approximately  20  percent  to  25  percent  of  funding  toward  early-stage  technology 
development,  with  the  Advanced  Technology  Program  as  one  of  the  principal  federal  programs 
focused  in  this  critical  area.  ATP  targets  technology  development  that  venture  capitalists  do 
not  address.  Venture  capitalists,  state  governments,  and  universities  contribute  only  between 
8 percent  and  16  percent  toward  early-stage  technology  development. 


PROJECT  AND  PORTFOLIO  ASSESSMENT 

Status  Reports  are  descriptive  mini-case  studies  for  each  completed  ATP  project,  written  several 
years  after  ATP  funding  ends.  Status  Reports  address  how  well  the  project  performed  against 
ATP’s  mission  objectives. 

A performance  rating  for  each  project  (zero  to  four  stars)  is  computed  using  a uniform  set  of  data. 
The  aggregation  of  stars  provides  a portfolio  view  of  ATP  performance.  For  example,  aggregating 
the  performance  ratings  for  the  first  120  ATP  projects  shows  the  following  distribution: 
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• 16  % 0 stars 

• 14  % 1 star 

• 29  % 2 stars 

• 31  % 3 stars 

• 10  % 4 stars 

The  largest  group  of  projects,  31  percent,  fell  into  the  three-star  category.  These  projects  show 
strong  progress.  Combining  these  with  the  10  percent  rated  outstanding  shows  an  impressive 
41  percent  of  projects  performing  at  a high  level.  Twenty-nine  percent  fell  into  the  two-star 
category.  These  projects  show  moderate  progress  but  are  not  particularly  robust  overall.  Thirty 
percent  of  the  projects  scored  one  star  or  less,  which  is  not  surprising,  given  that  ATP  projects 
are  high-risk  R&D  and  not  all  projects  are  expected  to  succeed.  Projects  may  fail  for  technical 
reasons,  business  reasons,  or  a combination  of  both. 

Each  status  report  also  includes  a patent  tree  for  each  patent  filed  during  or  after  the  ATP 
project  ends  to  show  the  citation  of  the  patent  in  subsequent  patents.  This  is  one  way  to  portray 
knowledge  spillovers. 


ANSWERING  THE  COUNTERFACTUAL  QUESTION:  WHAT  HAPPENS  WITHOUT  ATP? 

The  Survey  of  ATP  Applicants  was  conducted  in  1998,  2000,  and  2002.  This  survey  was  admin- 
istered to  all  applicants  in  the  previous  competition  year  in  order  to  compare  the  company  and 
project  characteristics  of  awardee  and  non-awardee  companies  soon  after  awards  are  announced. 
It  addresses  the  counterfactual  question,  what  happens  when  a project  does  not  receive  ATP 
funding.  The  Survey  of  ATP  Applicants  asks  organizations  that  did  not  receive  awards  what 
happened  to  their  projects  once  they  did  not  receive  funding  from  ATP.  The  survey  results  find 
that  41  percent  of  these  projects  are  not  pursued,  40  percent  are  pursued  on  a smaller  scale,  and 
of  those  pursued  on  a smaller  scale,  four  out  of  five  reports  that  the  project  scope  is  reduced  to 
below  40  percent  of  the  proposed  ATP  project. 

Evidence  from  the  Survey  of  ATP  Applicants  shows  that  ATP  is  successful  in  directing  funding 
to  projects  that  have  higher  technical  risk  and  longer  time  horizons  than  projects  proposed  by 
nonawardees  (ATP,  2003a).  A measure  of  technical  risk  is  the  probability  that  a project  will 
not  achieve  its  technical  goals. 

• Among  ATP  awardees,  the  average  estimate  for  the  probability  of  not  fully  achieving 
technical  goals  is  45  percent,  compared  to  nonawardees’  estimated  probability  of  31 
percent. 

• About  half  (54  percent)  of  ATP  awardees  expect  a time  horizon  of  4 years  or  more  on 
their  proposed  ATP  projects  compared  to  one-third  of  nonawardees. 

Proposed  ATP  projects  for  both  awardees  and  nonawardees  are  higher  risk  and  have  a longer 
time  horizon  than  to  ‘typical’  R&D  projects.  ATP  awardees  report  a greater  contrast  between 
their  proposed  and  typical  R&cD  projects,  compared  to  nonawardees. 
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A key  finding  is  that  ATP  awardees  attract  additional  funding  since  submitting  their  ATP 
proposal.  This  phenomenon  is  referred  to  as  the  halo  effect.  For  example,  three  out  of  four 
awardees  report  increased  funding,  while  one  out  of  four  non-awardees  reports  increased  fund- 
ing from  internal  company  sources.  ATP  awardees  are  also  more  likely  to  receive  funding  from 
external  sources.  One  out  of  three  awardees  reports  increased  funding,  and  only  one  out  of 
five  non-awardees  report  increased  funding  from  external  sources. 


MEASURING  ACCELERATION  EFFECTS 

The  Business  Reporting  System  (BRS)  allows  an  examination  of  ATP  awardees  from  a longitu- 
dinal perspective  and  from  a cross-sectional  perspective.  Responses  to  the  BRS  surveys  indicate 
that  ATP  funding  accelerated  RdcD  in  9 out  of  10  organizations.  Of  those  organizations  that 
indicated  they  were  ahead  in  their  RScD  cycle: 

• 13  percent  indicate  they  are  ahead  by  one  year. 

• 53  percent  indicate  that  they  are  ahead  by  one  to  three  years. 

® 7 percent  indicate  that  they  are  ahead  by  more  than  three  years. 

ATP  participants  report  that  the  acceleration  of  R&D  reduces  the  time  it  will  take  to  bring 
products  to  market  or  to  implement  new  production  processes.  Reduction  in  time-to-market  by 
two  years  or  more  is  anticipated  for  about  three  out  of  five  planned  commercial  applications.1 


MEASURING  OUTCOMES -BENEFIT-COST  STUDIES 

Benefit-cost  studies  are  one  of  the  primary  ways  to  measure  outcomes  quantitatively.  Outcomes 
are  difficult  to  measure  because  one  must  make  assumptions  about  the  impact  of  the  new  tech- 
nology and  acceptance  by  buyers.  The  results  of  individual  benefit-cost  studies  can  be  aggregated 
to  look  at  the  impact  (usually  prospective  estimates)  across  ATP.  The  net  social  benefits  from 
about  40  ATP  projects  are  estimated  to  be  $18  billion.  This  is  a more  than  adequate  return 
on  an  investment  of  $2.2  billion  that  ATP  has  made  between  1990  and  September  2004.  ATP 
projects  produce  large  benefits. 

One  example  of  a benefit-cost  study  is  Low-Cost  Manufacturing  Technology  for  Amorphous 
Silicon  Detectors,  a joint  venture  project  funded  by  ATP  in  1995  (Pelsoci,  2003).  Digital 
mammography  and  radiography  systems  are  innovative  technology  solutions  to  the  diagnostic 
and  productivity  limitations  of  conventional  x-ray  systems.  The  new  process,  which  was 
implemented  in  2004,  reduces  fabrication  costs  by  approximately  25  percent  without 
compromising  performance.  On  the  basis  of  33  million  mammography  and  68  million  chest 
x-rays  per  year,  estimated  benefits  are  $125  to  $193  for  every  $1  that  ATP  has  spent.  Societal 
benefits  include  avoidance  of  unnecessary  medical  procedures  as  a result  of  lower  false-positive 
rates,  improved  breast  cancer  detection,  reduced  patient  exposure  to  radiation,  and  reduced 
examination  time. 


1.  Based  on  Business  Reporting  System  (BRS)  survey  data  from  673  organizations  in  34/  ATP  projects  funded  from 
1993  to  1998,  for  projects  with  one  or  more  years  of  ATP  funding. 
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• 

• 

Recommendations  for  Future 
Directions  and  Conclusion 


ATP  is  continuously  improving  its  evaluation  program.  In  A Toolkit  for  Evaluating  Public  R&D 
Investment,  a list  of  recommendations  for  guiding  these  improvements  is  provided  (Ruegg  and 
Feller,  2003). 

• Increase  retrospective  analyses  based  on  market  data. 

• Incorporate  both  direct-  and  indirect-path  analysis  in  benefit-cost  case  studies, 
including  estimates  of  both  market  and  knowledge  spillovers. 

• Continue  status  reports  of  completed  projects  and  conduct  periodic  updates  for 
a sampling  of  projects. 

• Update  information  on  state  and  foreign  counterpart  programs. 

• Further  develop  several  of  the  promising  new  evaluation  techniques. 

• Deepen  analysis  of  knowledge  spillovers  beyond  patent  studies. 

• Identify  and  address  new  questions  that  arise  as  ATP  is  modified. 

• Pursue  analysis  of  failures  and  successes. 

• Continue  an  effective  mix  of  in-house  and  external  evaluation  studies. 

® Take  greater  advantage  of  evaluation  results  in  decision-making  processes. 

ATP  is  beginning  to  incorporate  these  recommendations  in  planning  future  work. 

ATP’s  multifaceted  and  integrated  evaluation  program  uses  widely  accepted  approaches  to 
economic  and  statistical  analysis  to  develop  estimates  of  impacts  of  ATP  funding  on  project 
timing  and  success.  In  this  paper,  we  have  discussed  the  best  practices  learned  from  our  15 
years  of  evaluation.  Through  the  use  of  these  evaluation  best  practices,  our  surveys,  studies, 
and  reports  show  that  ATP  is  indeed  meeting  its  mission  to  accelerate  the  development  of 
high-risk,  enabling  technologies. 
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■ 


About  the  Advanced  Technology  Program 


The  Advanced  Technology  Program  (ATP)  is  a partnership  between  government  and  private 
industry  to  conduct  high-risk  research  to  develop  enabling  technologies  that  promise  significant 
commercial  payoffs  and  widespread  benefits  for  the  economy.  ATP  provides  a mechanism  for 
industry  to  extend  its  technological  reach  and  push  the  envelope  beyond  what  it  otherwise 
would  attempt. 

Promising  future  technologies  are  the  domain  of  ATP: 

• Enabling  technologies  that  are  essential  to  the  development  of  future  new  and  sub- 
stantially improved  projects,  processes,  and  services  across  diverse  application  areas; 

• Technologies  for  which  there  are  challenging  technical  issues  standing  in  the  way  of 
success; 

• Technologies  whose  development  often  involves  complex  “systems”  problems  requiring 
a collaborative  effort  by  multiple  organizations;  and/or 

• Technologies  that  will  go  undeveloped  and/or  proceed  too  slowly  to  be  competitive  in 
global  markets  without  ATP. 

ATP  funds  technical  research,  but  it  does  not  fund  product  development  — that  is  the  domain  of 
the  company  partners.  ATP  is  industry  driven,  and  that  keeps  it  grounded  in  real-world  needs. 
For-profit  companies  conceive,  propose,  co-fund,  and  execute  all  of  the  projects  cost-shared  by 
x\TP;  these  projects  often  include  universities,  and  other  nonprofit  organizations. 

Smaller  firms  working  on  single-company  projects  pay  a minimum  of  all  the  indirect  costs 
associated  with  the  project.  Large  Fortune  500  companies  participating  in  single-company 
projects  must  pay  at  least  60  percent  of  total  project  costs.  Participants  of  joint  venture  projects 
pay  at  least  half  of  total  project  costs.  Single-company  projects  can  last  up  to  three  years;  joint 
venture  projects  can  last  as  long  as  five  years.  Companies  of  all  sizes  participate  in  ATP-funded 
projects.  To  date,  nearly  two  out  of  three  ATP  awards  have  gone  to  individual  small  businesses 
or  to  joint  ventures  led  by  a small  business. 

Each  project  has  specific  goals,  funding  allocations,  and  completion  dates  established  at  the 
outset.  Projects  are  monitored  and  can  be  terminated  for  cause  before  completion.  All  projects 
are  selected  in  rigorous  competitions  that  use  peer  review  to  identify  those  that  score  highest 
against  technical  and  economic  criteria. 

Contact  ATP  for  more  information: 

• On  the  Internet:  http://www.atp.nist.gov 

• By  e-mail:  atp@nist.gov 

• By  phone:  1-800-ATP-FUND  (1-800-287-3863) 

• By  writing  Advanced  Technology  Program,  National  Institute  of  Standards  and 
Technology,  100  Bureau  Drive,  Mail  Stop  4701,  Gaithersburg,  MD  20899-4701 


